TITLE OF THE INVENTION 
MOVING BODY DETECTING APPARATUS 

BACKGROUND OF THE INVENTION 
5 1) Field of the Invention 

The present invention relates to a technique of detecting a moving body by 
conducting various processing on an image picked up. 
2) Description of the Related Art 

As one of various conventional crime-prevention systems, there has been 

10 known a technique in which a moving body detecting apparatus detects a break-in 
body through the use of an image to make notification on the basis of the 
detection results. In general, this moving body detecting apparatus previously 
stores an image including no moving body as a background image and calculates 
a difference between the an image newly picked up and the previously stored 

1 5 background image for recognition of a moving body. Concretely, a difference 
between an image picked up and a background image is obtained to conducting 
threshold processing on a difference value for each pixel for producing a binary 
image so that a moving body is extracted through noise removal and labeling 
processing on the binary image. Moreover, a feature quantity of the moving 

20 body is calculated to make a decision as to whether or not the calculated feature 
quantity agrees with a predetermined judgmental criterion for making a decision 
as to whether that moving body is to be notified or not. In this case, the "feature 
quantity" signifies a physical quantity characterizing a moving body and, for 
example, there are "existence time" for which a moving body exists in an image, 

25 "area" of the moving body, "vertical and horizontal dimensions" of the moving 
body, "moving speed" at which the moving body moves, and others. 

Meanwhile, in a case in which a moving body detecting apparatus is used 
as an approach detecting apparatus for a vehicle, a shadow of a moving body, 
together with the moving body, can appear at a window portion of the vehicle due 
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to sunlight. For this reason, some moving body detecting apparatuses involves a 
technique of eliminating a shadow of a moving body from a picked-up image for 
precise detection of the moving body (for example, see Japanese Patent 
Laid-Open No. HEI 07-220049). Concretely, a difference between a background 
5 image and a background shifted image obtained by shifting the background image 
by a predetermined number of pixels horizontally and vertically is calculated to 
produce a background difference image which in turn, is binary -processed to 
produce a background binary image, moreover, as with the background image, 
with respect to a picked-up image obtained by taking a pickup (photograph) of a 

1 0 moving body, a pickup difference image is produced on the basis of a difference 
between a pickup image and a pickup shifted image obtained by shifting the 
pickup image by a predetermined number of pixels vertically and horizontally and 
the pickup difference image is binary-processed to produce a pickup binary image. 
Still moreover, the background binary image and the pickup binary image are 

15 exclusive-OR-processed to an exclusive-OR image, and the exclusive-OR image 
and the background binary image are AND-processed to remove the shadow of 
the moving body. 

In addition, as shown in FIG. 29, a passer or bicycle passing by a vehicle 
can be reflected in a window portion of the vehicle (see "803" in the illustration). 

20 FIG. 29 shows an image obtained by photographing a window portion (see "801" 
in the illustration) and a door portion (see "802" in the illustration) near a rear seat 
from its interior side. Still additionally, a shadow of a building due to sunlight, a 
reflection of head light of a vehicle in the nighttime, a shadow of a tree or the like 
can be reflected in the interior of a vehicle (see "804" in the illustration). For 

25 this reason, in some moving body detecting apparatuses, the aforesaid judgmental 
criterion is corrected on the basis of a feature quantity of each of things most 
extracted as moving bodies from a pickup image in the past (for example, 
Japanese Paten Laid-Open No. HEI 1 1-203567. For example, in the case of the 
aforesaid "area", if the judgmental criterion is set to be "area = 10000 pixels 
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(central value) ± 1000 pixels (width)", when moving bodies whose areas are 8000 
pixels to 10000 pixels are most extracted therefrom, the aforesaid judgmental 
criterion is corrected to "area = 9000 pixels (central value) ± 1000 pixels (width)". 
Incidentally, likewise, the correction is made with respect to the other judgmental 

5 criteria such as "existence time". 

However, even if, as mentioned above, the shadow of a moving body is 
removed or the judgmental criterion is corrected on the basis of a feature quantity 
of a moving body, there is a possibility that the detected feature quantity of a 
moving body, such as a passer or shadow of a building, agrees with a judgmental 

10 criterion. In this case, the aforesaid moving body detecting apparatus is at risk of 
detecting that moving body as a body to be notified in error. Moreover, in a case 
in which the aforesaid judgmental criterion is set in advance or through correction 
on the basis of a feature of the whole body of a thief, for example, when only the 
upper half of the body or arm of the thief is reflected in a window portion of a 

15 vehicle, there is a possibility that the feature quantity of that thief can disagree 
with the aforesaid judgmental criterion and, in this case, the moving body 
detecting apparatus cannot detect the thief as a body to be notified. It can be 
considered that this applies likewise to moving body detecting apparatuses 
designed to detect a body to be notified on the basis of a feature quantity of a 

20 moving body. That is, for example, in the case of preventing a person 
committing a theft (which will be referred to hereinafter as a "thief from 
breaking in the interior of a vehicle, both a passer and the thief are a human being 
and, hence, the feature quantities thereof becomes similar to each other. This is 
because the feature of the passer agrees with the judgmental criterion set for the 

25 purpose of detecting the thief. 

Furthermore, the employment of such a moving body detecting apparatus 
is not limited to vehicles but being also applicable to a spot such as an entrance or 
window of a building requiring the prevention of the break-in of a thief or a place 
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such as a store or art museum requiring the prevention of approach of a thief 
around an article or picture. 

SUMMARY OF THE INVENTION 
5 The present invention has been developed in consideration of these 

situations, and it is therefore an object of the invention to provide a technique of 
detecting only a body to be notified without detecting a body similar in feature to 
the body to be notified. 

Another object of the present invention is to provide a moving body 
10 detecting apparatus capable of accurately detecting a moving body which can fall 
into a background like a small moving body in a case in which a picked-up image 
does not have a sufficient contrast or if much noise is included in the picked-up 
image. 

A further object of the present invention is to provide a displacement data 
1 5 extracting method and a body detecting apparatus capable of associating the same 
image pickup points shown in different images with a small calculation quantity 
to displacement data indicative of the displacement with high accuracy. 

In accordance with a first aspect of the present invention, there is provided 
a moving body detecting apparatus characterized by detecting a body moving 
20 from an area permitting the existence of an body to an area inhibiting the 

existence of an body. Concretely, image, acquiring means first acquires static 
images in an identical monitoring range continuing in time sequence. 
Subsequently, decision processing means receives a plurality of static images 
continuing in time sequence from the image acquiring means to make a 
25 comparison among these static images for making a decision as to whether or not 
a body moves from an "allowable area" set in advance in the monitoring range 
and allowing the existence of a body to an "unallowable area" set in advance in 
the monitoring range and inhibiting the existence of a body. The image 
acquiring means is realizable through the use of a photographing device such as a 
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CCD camera or infrared camera, or it is also realizable with an interface device 
which obtains images picked up by a device separate from the moving body 
detecting apparatus. 

Thus, since this moving body detecting apparatus is made to detect only a 
5 body moving the allowable area to the unallowable area as a body to be notified, 
for example, only a body to be notified is detectable without detecting an body 
similar in feature to the object to be notified, such as a passer or the like to a thief 

As an example, a description of the aforesaid allowable area and 
unallowable area will be given hereinbelow in the case of a moving body 

10 detecting apparatus used as a break-in detecting apparatus for a vehicle. The 

image acquiring means is set so as to take a photograph of a door from the interior 
side of the vehicle. For example, in a case in which a person committing a theft 
(which will be referred to hereinafter as a "thief) puts his/her hand through a 
window in the interior of the vehicle for stealing a purse or the like therefrom, the 

15 hand of the thief moves from a window portion in static images to a door body 
portion (a portion other than the window) therein. Therefore, the window 
portion of the door in the static images is set as the "allowable area" while the 
door body in the static images is set as the "unallowable area". In this case, the 
decision processing means makes a decision that the hand of the thief moving 

20 from the window portion to the door body portion in the static images is a body 
moving from the allowable area to the unallowable area. However, in a case in 
which a thing put in the interior of the vehicle is a large one such as a handbag 
and the body of the handbag appears in the door body portion in the static images 
and a handle of the handbag (which will be referred to simply as a "handle") 

25 appears in the window portion in the static images. In this case, the thief can 
grasp the handle and brings the handbag into the exterior of the vehicle through 
the window thereof without moving the hand from the window portion to the door 
body portion in the static images. At this time, the body of the handbag is 
shifted from the "unallowable area" to the "allowable area". Therefore, it is 
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considered that, in the static images, the window portion is set as the "unallowable 
area" while the door body portion is set as the "allowable area". In this case, 
when the thief grasps the handle and brings the handbag out into the exterior of 
the vehicle, the decision processing means makes a decision that the body of the 
5 handbag moving from the door body portion to the window portion in the static 
images is an body moving from the allowable area to the unallowable area. 

In this case, the aforesaid allowable area and unallowable area can be set 
as a point set such as bit map data. This eliminates the need to make a 
calculation for each pixel for making a decision as to whether or not each of the 

10 pixels of a difference image exists in an allowable are or the like, thus reducing 
the burden of the decision processing means. Moreover, it is also appropriate 
that the allowable area and the unallowable area are prescribed by the boundary 
lines of their outer circumferences. This enables the quantity of set data such as 
the allowable area to become only data for setting the boundary lines. This 

15 quantity decreases as compared with the data quantity in the case of the allowable 
area and others being set as a point set, which reduces the memory size of a 
storage unit for storing these set data. Incidentally, in the case of expressing the 
aforesaid boundary lines with a function, the quantity of the set data such as the 
allowable area is reducible. If the aforesaid boundary line is a straight line, the 

20 data quantity is further reducible in a manner such that the allowable area and 
others are set with only both end points of the straight line. 

Meanwhile, if a moving body has a lower luminance as compared with a 
background, there is a possibility that the moving body is broken by the 
background. For example, in the case of a monochrome image, when the 

25 luminance value of the background is high and the luminance value of the moving 
body is low, even if the moving body exists at a given pixel, the luminance value 
of the background achieves domination over the luminance value of the moving 
body when the moving body moves and, hence, the moving body can disappear as 
a result. Therefore, in the aforesaid decision processing, it is considered that the 



% 



7 



decision processing means calculates an absolute value of the difference in 
luminance value between each of a plurality of static images from the image 
acquiring means and the background image stored in advance for each pixel to 
produce a plurality of difference images, and makes the comparison between the 
5 difference images to make a decision as to whether a body moving from the 

allowable area to the unallowable area exists or not. The "background image" 
signifies an image in which a moving body is absent and the "luminance value" 
may signify a simple brightness in the case of a monochrome image and, in the 
case of a color image, may signify an independent luminance value of each of 

10 RGB colors or the result luminance value of the three colors. 

This difference image is produced by calculating the absolute value of the 
difference in luminance value between a static image received from the image 
acquiring means and a background image stored in advance and, hence, the 
difference image does not represent the comparison result in magnitude of 

15 luminance between both the images (which of the images is higher in luminance) 
but signifying the comparison result in the difference of luminance therebetween 
(the degree of the difference of luminance therebetween). Therefore, this 
eliminates the above-mentioned problem of a moving body being broken because 
the background luminance value is high, thereby enabling high-accuracy detection 

20 of a moving body. 

Moreover, it is also appropriate that the moving body detecting apparatus 
further comprises preliminary decision processing means for making a 
comparison with respect to an allowable area between a background image stored 
in advance and a static image from the image acquiring means to make a 

25 preliminary decision as to whether or not the difference therebetween exceeds a 
predetermined value, and the decision processing means carries out the decision 
processing only when the preliminary decision result in the preliminary decision 
processing means shows the difference exceeding the predetermined value. In 
this case, the "predetermined value" signifies a quantitatively calculable physical 



8 



quantity, for example, the number of pixels having different luminance values 
(disagreement in luminance) or the discrete degrees of these pixels. 

As an example utilizing this arrangement, a description will be given 
hereinbelow of a case in which the moving body detecting apparatus is employed 
as a break-in detecting apparatus for a vehicle. First of all, a window portion in 
static images is set as an allowable area. Then, the preliminary decision 
processing means makes a comparison between the window portion (allowable 
area) of a background image stored in advance and the window portion (allowable 
area) of a static image from the image acquiring means to continuously make a 
decision as to whether or not a difference indicating the existence of a person 
exists between both the images. Following this, the decision processing means 
implements a series of decision processing only while the preliminary decision 
processing means continues the decision indicative of "a difference to which a 
person exists". 

Thus, for example, only when a person is reflected in the window portion, 
that is, only when the person exists in the vicinity of a door in the exterior of the 
vehicle, the decision means carries out the above-mentioned series of decision 
processing. Therefore, the decision processing is conducted only when needed, 
and if the power per unit time to be used for the decision processing in the 
decision processing means is larger than the power per unit time to be used for the 
preliminary decision in the preliminary decision processing means, the power 
consumption per unit time is reducible as compared with the case in which the 
decision processing means implements the decision processing at all times. 

In this connection, in the decision processing means, as a concrete manner 
of making a decision as to whether a moving body exists within a difference 
image, it is also appropriate to use a method in which a difference image is 
converted into binary image data and labeled so that, on the basis of information 
on area and position of the resultant pixel set, a decision is made as to whether a 
moving body exists or not. In this case, for example, the decision based on the 
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"area and position of the pixel set" signifies that, of the pixel sets, a pixel set(s) 
whose area exceeds a predetermined value is selected so that, on the basis of 
whether or not the positional information on that pixel set indicates movement, a 
decision is made as to whether a moving body exists or not. This enables 
5 suppressing the influence of noises residing in an image acquired by the image 
acquiring means, thereby achieving high-accuracy detection on the existence of a 
moving body. 

Moreover, it is also appropriate that, at the labeling, the area of the pixel 
set is simultaneously calculated through the use of a counter. This enables a 

10 more efficient decision as to whether or not a moving body exists in the image. 

Still moreover, it is also appropriate that the decision processing means 
calculates a vector indicative of a motion of the pixel set on the basis of the 
positional information on the pixel set. This can reduce the data quantity as 
compared with the use of the pixel set and, hence, achieves the decision 

15 processing faster as compared with the decision processing based on the pixel set. 

Yet moreover, it is also appropriate that, when a decision is made that a 
moving body exists, the decision processing means transmits the information on 
the existence of the moving body through a transmitting means to an external 
different apparatus such as a portable telephone pertaining to the owner of the 

20 vehicle or a telephone in a house of the owner of the vehicle. This achieves 

quick notification on the existence of an intruder from the exterior of the vehicle 
to the interior thereof to the owner of the vehicle. 

In addition, it is also appropriate that the moving body detecting apparatus 
further comprises notifying (or broadcasting) means for notifying the existence of 

25 a moving body to a user, and the decision processing means issues a command to 
the notifying means to notify the existence of the moving body to the user when 
detecting the moving body existing in a monitoring range. As concrete examples 
of the notifying means, there are 1) a display is made on an indicator, 2) a 
notification is made through the use of a voice, 3) a lamp is lighted, 4) a buzzer is 
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rung, and others. This enables the moving body detecting apparatus itself to 
notify the existence of a moving body to a user. 

Still additionally, it is also appropriate that the moving body detecting 
apparatus further comprises warning means for warning a moving body, and the 
decision processing means warns the moving body when detecting that the 
moving body continuously exists in a monitoring range for a predetermined 
period of time. As the concrete examples of the warning means, there are 1 ) a 
notification is made through the use of a voice, 2) an alarm lamp is lighted, 3) a 
buzzer is rung, and others. This can prevent the break-in-vehicle of a moving 
body. 

Yet additionally, when making a decision that the moving body exists 
therein, the decision processing means preserves an image of the moving body in 
an image storage means. This can store the fact that valuables such as a purse 
were stolen from the interior of the vehicle. 

Preferably, the moving body detecting apparatus can also be used as an 
antitheft apparatus. Since the consumable power is limited during the vehicle 
storage, difficulty can be experienced in locating a sufficient light source(s), thus 
making it difficult to provide a static image having a sufficient contrast. Even in 
such a situation, the moving body detecting apparatus according to the present 
invention can achieve the detection with high accuracy, thus providing a high 
utility value. 

According to the present invention, the decision processing means of the 
moving body detecting apparatus is also realizable in a manner such that a 
program functioning as the decision processing means is used and implemented in 
a computer incorporated into the moving body detecting apparatus. This 
program is recorded on a computer-readable recording medium such as a flexible 
disk, magnetic optical disk, CD-ROM, hard disk, ROM or RAM, and is loaded 
into the computer and activated when needed. It is also possible to perform the 
loading and activation through a network. 
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BRIEF DESCRIPTION OF THE DRAWINGS 

Other objects and features of the present invention will become more 
readily apparent from the following detailed description of the preferred 
embodiments taken in conjunction with the accompanying drawings in which: 

FIG. 1 is a block diagram showing an internal configuration of a moving 
body detecting apparatus according to a first embodiment of the present invention; 

FIG. 2A is an illustration of a static image obtained by taking a photograph 
through the use of a camera; 

FIG. 2B is an illustration useful for explaining a monitoring range to be set 
in a static image; 

FIG. 2C is an illustration useful for explaining a monitoring range to be set 
in a static image; 

FIGs. 3 A to 3 C are illustrations useful for explaining processing for 
producing a difference image, and FIG. 3A shows a background image, FIG. 3B 
shows a present image, and FIG. 3C shows a difference image; 

FIGs. 4A to 4C are illustrations useful for explaining the binary 
conversion and labeling processing as image processing, and FIG. 4A shows a 
background image, FIG. 4B shows a present image, and FIG. 4C shows a 
difference image further undergoing the binary conversion and labeling 
processing; 

FIG. 5 A is an illustration useful for explaining the processing for detecting 
a motion of a pixel set on the basis of two pixels continuing in time sequence; 

FIG. 5 B is an illustration useful for explaining the processing for detecting 
a motion of a pixel set on the basis of two pixels continuing in time sequence; 

FIG. 6 is an illustration useful for explaining decision processing to be 
conducted in a recognizing unit; 

FIG. 7 is a flow chart showing decision processing according to the first 
embodiment; 
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FIG. 8 is a block diagram showing another internal configuration of the 
moving body detecting apparatus according to the first embodiment; 

FIG. 9 is a flow chart useful for explaining decision processing to be 
implemented; 

FIGs. 10A and 10B are illustrations for explaining the decision processing; 

FIG. 1 1 is an illustration of a background image for explaining the 
decision processing; 

FIGs. 12 A and 12B are illustrations of present images for explaining the 
decision processing; 

FIGs. 13A and 13B are illustrations of difference images for explaining 
the decision processing; 

FIG. 14 is an illustration of a composite image for explaining the decision 
processing; 

FIG, 15 is a block diagram showing a further configuration of the body 
detecting apparatus according to the first embodiment of the present invention; 

FIGs. 16A and 16B are illustrations useful for explaining a method of 
mounting an image pickup device; 

FIGs. 17A and 17B are illustrations of an input image at image composing 
and an example of an image of an edge forming a starting point of a trajectory, 
extracted from the input image; 

FIGs. 1 8A to 1 8C are illustrations useful for explaining an input image, an 
example of a composite image and a trajectory extracting method; 

FIG. 19 is a flow chart useful for explaining preliminary decision 
processing according to a third embodiment of the present invention; 

FIG. 20 is a flow chart useful for explaining decision processing according 
to the third embodiment; 

FIG. 21 is a block diagram showing a configuration of an image 
composing unit of an object detecting apparatus according to a fifth embodiment 
of the present invention; 
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FIGs. 22A to 22C are illustrations useful for explaining an operation of an 
FOE (Focus Of Expansion) displacement compensating unit when an advancing 
direction of vehicle is changed; 

FIGs. 23 A and 23B are illustrations useful for explaining an operation of 
5 an FOE displacement compensating unit when the posture of a vehicle is inclined 
in a roll direction; 

FIGs. 24A and 24B are illustrations useful for explaining a method of 
mounting an image pickup device; 

FIGs. 25 A to 25C are illustrations useful for explaining a method of 
1 0 mounting an image pickup device; 

FIGs. 26A and 26B are illustrations useful for explaining a method of 
mounting an image pickup device; 

FIGs. 27A and 27B are illustrations useful for a difference in effect 
between a conventional apparatus and the present invention; 
15 FIGs. 28 A and 28B are illustrations useful for explaining a method of 

detecting a trajectory length; 

FIG. 29 is an illustration for explaining conventional decision processing; 

DESCRIPTION OF THE PREFERRED EMBODIMENTS 
20 Embodiments of the present invention will be described hereinbelow with 

reference to the drawings. 
First Embodiment 

(Configuration of Moving Body Detecting Apparatus) 

FIG. 1 is a block diagram showing an internal configuration of a moving 
25 body detecting apparatus according to a first embodiment of the present invention. 
The moving body detecting apparatus, generally designated at reference numeral 
1 1, is made to function as an antitheft apparatus. As FIG. 1 shows, the moving 
body detecting apparatus 1 1 is made up of a camera 13, an area selecting unit 15, 
a difference calculating unit 21, a binary conversion/labeling unit 25, a feature 
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calculating unit 29, a motion detecting unit 33, a recognizing unit 37, a control 
unit 39, a background image memory 17, a present image memory 19, a 
difference image memory 23, a binary conversion/labeling memory 27, a feature 
calculation memory 31, a motion detection memory 35 and a speaker 41. 

Of these components, each of the area selecting unit 15, the difference 
calculating unit 21, the binary conversion/labeling unit 25, the feature calculating 
unit 29, the motion detecting unit 33, the recognizing unit 37 and the control unit 
39 is constructed as an electronic circuit including a CPU, a ROM, a RAM and 
others. On the other hand, each of the background image memory 17, the 
present image memory 19, the difference image memory 23, the binary 
conversion/labeling memory 27, the feature calculation memory 3 1 and the 
motion detection memory 35 is constructed with a semiconductor memory such as 
a DRAM or SDRAM. Each of these memories can individually be constructed 
with a semiconductor memory, or all of these memories can also be constructed 
with one or plural semiconductor memories. 

(Configuration of each of parts of Moving Body Detecting Apparatus) 

The camera 13 is attached to a pillar or the like in the interior of a vehicle 
and is set to take a photograph continuously in time sequence in a state where 
break-in-possible places such as doors for a driver's seat and assistant driver's 
seat and their vicinities, right- and left-hand doors for a rear seat and their 
vicinities, a windshield and its vicinity and a rear windshield and its vicinity are 
set as a monitoring region or range. FIG. 2 A shows a static image of the 
assistant driver's seat door and the rear seat left-hand door and their vicinities, 
obtained through the use of the camera 13. Moreover, the camera 13 has a 
performance to pick up 30 frames per second as monochrome static images. 
This camera 13 is realizable with a photographing device such as a CCD camera 
or infrared camera, or it can also be an interface device which only acquires 
images taken by a device separate from the moving body detecting apparatus. 



The area selecting unit 15 has a function to divide the static images from 
the camera 13 into an interior image of a vehicle and an exterior image thereof 
and stores them in the background image memory 17 and the present image 
memory 19. This area selecting unit 15 stores the contents of a monitoring 
region set in order to confirm as to whether an intruder is included in static images 
from the camera 13. In this embodiment, monitoring regions (see regions "201" 
and "203" in FIG. 2B) for monitoring window portions in the static images are set 
as "allowable areas" which permit a body to exist, while monitoring regions (see 
regions "202" and "204" in the same illustration) for monitoring door body 
portions in the static images are set as "unallowable areas" which do not permit a 
body to exist. In this case, the assistant driver's seat door window portion and 
the door body portion depend upon the positional relation in a height direction of 
the vehicle. For example, in the image, the region "201" forming the allowable 
area and the region "202" forming the unallowable area have a positional 
relationship in the vehicle height direction (the region "201" is positioned above 
the region "202"). Moreover, the assistant driver's seat door window portion 
and the rear seat left-hand door have a positional relationship in a longitudinal 
direction of the vehicle. For example, in the image, the region "201" and the 
region "203", forming the allowable areas, has a positional relationship in a 
longitudinal direction (the region "201" is positioned on the left side). In the 
description of this embodiment, the "vehicle height direction" of the vehicle will 
be referred to as a "vertical direction" in each image while the "longitudinal 
direction" of the vehicle will be referred as to a "horizontal direction" in each 
image. Moreover, each of the above-mentioned monitoring regions is set in the 
form of a point set such as bit map data. 

The difference calculating unit 21 has a function to produce a difference 
image on the basis of a "background image" stored in the background image 
memory 17 and a "present image" stored in the present image memory 19 and to 
store the produced difference image in the difference image memory 23. 
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Concretely, the difference calculating unit 21 reads out a background image (see 
FIG. 3 A) from the background image memory 17 and a present image (see FIG. 
3B) from the present image memory 19 to, with respect to each pixel, calculate 
the absolute value of a difference in luminance between both the images for 
5 producing a difference image in a state where the absolute value is used as a pixel 
value (see a pixel set "301" in FIG. 3C). The produced difference image is put 
in the difference image memory 23. 

The binary conversion/labeling unit 25 has a function to implement 
various image processing including processing conversion into binary image data 

1 0 and labeling and store the resultant image in the binary conversion/labeling 

memory 27. The feature calculating unit 29 has a function to calculate a feature 
of each of pixel sets in an image stored in the binary conversion/labeling memory 
27 and to store the calculation result, together with the image, in the feature 
calculation memory 3 1 . 

1 5 The motion detecting unit 33 has a function to detect a motion of each of 

the pixel sets on the basis of the image and each pixel set feature stored in the 
feature calculation memory 3 1 and to store the detection result, together with the 
feature of the each of the pixel sets, in the motion detection memory 35. 
Concretely, the motion detecting unit 33 reads out two new images continuing in 

20 time sequence and the feature calculation results of image sets included in the 
respective images from the feature calculation memory 3 1 . Then, as shown in 
FIG. 5 A, on the basis of areas of the pixel sets, the motion detecting unit 33 
makes a detection as to whether or not a pixel set existing in the old image (which 
will be referred to hereinafter as a "previous frame"; see FIG. 5A) of the aforesaid 

25 two images also exists in the new image (which will be referred to hereinafter as a 
"present frame"; see FIG. 5A) of the aforesaid two images. For example, in a 
case in which the object of detection is limited to the upper half of a thief or the 
like reflected in a window of the vehicle, the detection can be made on the basis 
of the area of the object. Moreover, if a pixel set is detected which exists in 
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common to the previous frame and the present frame, the motion detecting unit 33 
calculates, as a motion vector, a movement of the position of the center of gravity 
of the pixel set between the previous frame and the present frame. Incidentally, 
it is also appropriate that the two movements of the position of the center of 
5 gravity are indicated by distances in X and Y directions. As one example, as 

shown in FIG. 5B, if a pixel set "511" moves from a position indicated by a dotted 
line to a position indicated by a solid line, the center of gravity "514" in the 
previous frame moves to the center of gravity "513" in the present frame. The 
moving amount of the center of gravity at this time is indicated by the length of an 

10 arrow "517". Moreover, this also applied to a pixel set "512". Lastly, the 

information indicative of the motion vector, together with the image and the pixel 
set feature, is stored in the motion detection memory 35. 

The recognizing unit 37 has a function to, on the basis of the image and 
the feature/motion of each of the pixel sets stored in the motion detection memory 

15 35, recognize whether or not a moving body exists in a pixel set. Concretely, the 
recognizing unit 37 reads out the image and the information indicative of the 
feature/motion of the pixel set from the motion detection memory 35. 
Subsequently, on the basis of the images and various information, the recognizing 
unit 37 makes a decision as to whether a pixel set satisfying all the following 

20 conditions (1) to (3) exists or not, thereby making a decision as to whether an 
intruder breaking from the exterior of a vehicle into the interior of the vehicle 
exists or not. 

Condition (1) : in monitoring regions (the monitoring regions "201" and 
"203" in FIG. 2B), if the following two conditions (1-1) and (1-2) are 
25 simultaneously satisfied, a decision is made that the upper half of a thief is 
reflected in a window portion of a vehicle. 

the area of a pixel set > a first threshold — (1-1) 

the length of a motion vector < a second threshold — (1-2) 
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The first threshold corresponds to an area of a pixel set determined to 
detect the upper half of a thief reflected in a window portion of a vehicle and is 
prescribed in advance through experiments or the like. The second threshold 
corresponds to a length of vector set to detect a motion of the upper half of a thief 
5 which stops in the vicinity of the vehicle and appears in a window portion of the 
vehicle but set so as not to detect the upper half of a passer reflected in the 
window portion of the vehicle and is prescribed in advance through experiments 
or the like. 

Condition (2) : in monitoring regions (the monitoring regions "202" and 
10 "204" in FIG. 2B), if the following two conditions (2-1) and (2-2) are satisfied, a 
decision is made that an arm of a thief or the like comes from a window portion of 
a vehicle to a door body portion thereof. 

the area of a pixel set > a third threshold — (2-1) 
the direction of a motion vector is downward — (2-2) 
15 The third threshold corresponds to an area of a pixel set determined to 

detect a hand or an arm of a thief reflected in the door body portion of the vehicle 
and is prescribed in advance through experiments or the like. 

Condition (3) : When a pixel set in the exterior of the vehicle and a pixel 
set in the interior thereof overlap with each other in a horizontal direction, a 
20 decision is made that the pixel set residing in the exterior of the vehicle and the 
pixel set residing in the interior thereof are integral with each other. 

Explaining with reference to FIG. 6, in a case in which each of pixel sets 
"601", "602" and "603" satisfies the above-mentioned conditions (1) and (2), the 
width (see an arrow) of the pixel set "601" in a horizontal direction and the width 
25 (see an arrow) of the pixel set "603" in the horizontal direction do not overlap 

with each other as the positional relationship in the horizontal direction. On the 
other hand, the width (see an arrow of the pixel set "602" in the horizontal 
direction and the width of the pixel set "603" in the horizontal direction overlap 
with each other as the positional relationship in the horizontal direction. 
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The speaker 41 is composed of a horn or the like and has a function to 
issue an alarm sound for notifying the existence of an intruder from the exterior of 
the vehicle to the interior thereof to a passenger such as a driver and to issue a 
warning sound to the body existing continuously in the monitoring region, which 
5 is set to monitor the exterior of the vehicle, for a predetermined period of time. 
(Configuration of Each of Memories) 

The background image memory 17 is a memory for storing an image 
(background image) including no moving body forming an object of detection. 
This background image can be an image picked up by the camera 13 immediately 
10 after the power-on in the moving body detecting apparatus 1 1, or it can also be, of 
the images taken by the camera 1 3 immediately after the power-on, an image 
selected periodically as an image which does not vary in time sequence. 
Moreover, an image taken in advance through an manipulation by a user is also 
acceptable. 

15 The present image memory 19 is for storing the present image. The 

difference image memory 23 is for storing a difference image, and the binary 
conversion/labeling memory 27 is for storing an image which has undergone 
conversion-into-binary and labeling processing. Moreover, the feature 
calculation memory 3 1 is for storing an image which has undergone the feature 

20 calculation, and the motion detection memory 35 is for storing an image which 
has undergone the motion detection. Each of these memories is internally 
divided into a region (which will be referred to hereinafter as a "vehicle interior 
region") for storing an interior portion including a door portion of a background 
image or a present image and a region (which will be referred to hereinafter as a 

25 "vehicle exterior region") for storing an exterior portion including a window 
portion of a background image or a present image. 
(Description of Control Unit 39] 

The control unit 39 has a function to generally control the area selecting 
unit 15, the difference calculating unit 21, the binary conversion/labeling unit 25, 
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the feature calculating unit 29, the motion detecting unit 33 and the recognizing 
unit 37. 

In this embodiment, the camera 13 corresponds to the image acquiring 
means, and area selecting unit 15, the difference calculating unit 21, the binary 
conversion/labeling unit 25, the feature calculating unit 29, the motion detecting 
unit 33, the recognizing unit 37 and the control unit 39 correspond to the decision 
processing means, and the speaker 41 corresponds to the notifying means and the 
warning means. 

(Description of Decision Processing) 

Referring to a flow chart of FIG. 7, a description will be given 
hereinbelow of the decision processing to be implemented in the control unit 39. 
The implementation of this decision processing starts in response to the power-on 
in the moving body detecting apparatus 1 1 . 

First of all, in SI 05 (step 105), an instruction is given to the area selecting 
unit 15 for initializing the background memory 17 and the present image memory 
19. Upon receipt of this instruction, the area selecting unit 15 initializes the 
background memory 17 and the present image memory 19. After the completion 
of the initialization, the area selecting unit 15 makes a reply on the fact of the 
initialization completion to the control unit 39, and upon receipt of this reply, the 
control unit 39 executes the following steps. Although the description will be 
omitted in the following steps, as with this step, when receiving various types of 
instructions, each of the units (the area selecting unit 15, and others) makes a 
replay on the completion of that instruction to the control unit 39, and upon 
receipt of the replay, the control unit 39 executes the following steps. 

In the next SI 10, an instruction is given to the area selecting unit 15 for 
receiving static images from the camera 13 and storing them in the background 
image memory 17. In response to this instruction, the area selecting unit 15 puts 
the following background images in the background image memory 17. That is, 
when receiving the static images of the assistant driver's seat/rear seat door 
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portion/window portion in a vehicle from the camera 13 as shown in FIG. 2 A, the 
area selecting unit 1 5 encodes the pixel signals of portions corresponding to 
monitoring regions in the static images with a numerical value "1" (portions 
indicated by white color) and encodes the pixel signals of the other areas with a 
5 numerical value "0" (portions indicated by black color), as shown in FIG. 2B. 
Then, the area selecting unit 1 5 calculates two-dimensional coordinates of the 
pixel signals encoded with the numerical value "1" and makes judgment on the 
position of each of the pixel signals in the vehicle interior area or the vehicle 
exterior area. Moreover, the area selecting unit 1 5 stores, as a background image, 

10 each of the pixel signals in the relevant one of the vehicle interior area and vehicle 
exterior area of the background image memory 17. 

Subsequently, in SI 15, an instruction is given to the area selecting unit 15 
for receiving a static image from the camera 15 to store it as the present image in 
the present image memory 19. Upon receipt of this instruction, the area 

15 selecting unit 15 stores the following present image in the present image memory 
19. That is, when receiving a static image from the camera 13, the area selecting 
unit 1 5 calculates two-dimensional coordinates of each of pixel signals without 
encoding each of the pixel signals of the static image, and makes judgment on the 
position of each of the pixel signals in the vehicle interior area or the vehicle 

20 exterior area to put it as the present image in the relevant one of the vehicle 
interior area and the vehicle exterior area of the present image memory 19. 
Although the description will be omitted in the following steps, as with this step, 
when receiving various types of instructions, each of the units (the area selecting 
unit 15, and others) reads out an image from the corresponding memory and stores 

25 an image in the corresponding memory according to the selection between vehicle 
interior area and the vehicle exterior area. 

Following this, in S 120, an instruction is given to the difference 
calculating unit 2 1 for producing a difference image. Upon receipt of this 
instruction, with respect to each of pixels, the difference calculating unit 21 
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calculates the absolute value of the difference in luminance value between a 
background image (see FIG. 4 A) stored in the background image memory 17 and 
a present image (see FIG. 4B) stored in the present image memory 19 and 
produces a difference image in a manner such that this value is used as a pixel 
value. The produced difference image is put in the difference image memory 23. 

In S125, an instruction is given to the binary conversion/labeling unit 25 
for conducting the image processing on the difference image. Upon receipt of 
this instruction, the binary conversion/labeling unit 25 first converts the difference 
image, stored in the difference image memory 23, into binary image data 
according to luminance value. In this case, the "luminance value" signifies a 
simple brightness in the case of a monochrome image and, in the case of a color 
image, signifies an independent luminance value of each of RGB colors or the 
result luminance value of the three colors. Moreover, each of sets of pixel 
signals, expressed by the same value, is labeled (the pixel sets "40 1" to "403" in 
FIG. 4C). The image labeled for each pixel set is stored in the binary 
conversion/labeling memory 27. 

In SI 30, an instruction for the calculation of a feature of each of the pixel 
sets is given to the feature calculating unit 29. Upon receipt of this instruction, 
the feature calculating unit 29 first reads out an image from the binary 
conversion/labeling memory 27 and then calculates the area, center of gravity and 
average luminance of each of the pixel sets in the image read out therefrom. 
Moreover, the feature calculating unit 29 calculates the coordinates of a upper left 
corner and lower right corner of an external-shape rectangular configuration of the 
pixel set. These calculation results, together with the image, are stored in the 
feature calculation memory 3 1 . 

In SI 35, a decision is made as to whether or not two or more images 
different in time sequence are stored in the feature calculation memory 31. If 
two or more images different in time sequence are not stored in the feature 
calculation memory 31 (S135: NO), the operational flow returns to SI 15 to 
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implement the steps S115toS130 repeatedly. Incidentally, in a case in which 
the operational flow returns to SI 15 to repeat SI 15 to SI 30, the older one of the 
images stored in each of the memories other than the background image memory 
1 7 can be deleted. On the other hand, if two or more images different in time 
5 sequence are stored in the feature calculation memory 31 (SI 35: YES), the 
operational flow advances to SI 40, 

In SI 40, an instruction is given to the motion detecting unit 33 for 
detecting a motion of pixel sets of the image. In response to this instruction, the 
motion detecting unit 33 first reads out two images continuing in time sequence 

10 and a feature calculation result of a pixel set included in each of the images from 
the feature calculation memory 3 1 and then, on the basis of the area of a pixel set, 
detects whether or not a pixel set residing in the previous frame also exists in the 
present frame. If a pixel set residing in common to the previous frame and the 
present frame is not detected (SI 40: NO), the operational flow returns to SI 15 to 

1 5 conduct S 1 1 5 to S 1 30 repeatedly. 

On the other hand, in the case of the detection of a pixel set residing in 
both the previous frame and present frame (S140: YES), the movement of the 
position of the center of gravity of the pixel set between the previous frame and 
the present frame is calculated as a "motion vector". Lastly, information 

20 indicative of this motion vector, together with the image and the information 
representative of the feature of the image set, is stored in the motion detection 
memory 35. 

In SI 45, an instruction is given to the recognizing unit 37 for 
implementing the recognition processing. In response to this instruction, the 
25 recognizing unit 37 reads out the image and the information indicative of the 

feature and motion of the pixel set from the motion detection memory 35 and, on 
the basis of these information, makes decision as to whether or not there is a pixel 
set satisfying all the above-mentioned conditions (1) to (3), thereby making a 
decision as to whether an intruder from the exterior of the vehicle to the interior 
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thereof exists or not, with this decision result being sent as a reply to the control 
unit 39. 

The operational flow from SI 50 depends on the decision result in the 
recognizing unit 37. That is, if the decision result indicates that an intruder from 
5 the exterior of the vehicle to the interior thereof exists, the operational flow goes 
to SI 55. On the other hand, if the decision result indicates that an intruder from 
the exterior of the vehicle to the interior thereof does not exist, the operational 
flow returns to SI 15 to repeat SI 15 to S 130. 

In SI 55, a signal is sent to the speaker 41 to ring a horn for a 

10 predetermined (constant) period of time (for example, 1 minute). Other manners, 
such as lighting a lamp or sounding a siren or buzzer, are also acceptable. After 
the elapse of the predetermined period of time, the sending of a signal to the 
speaker 41 comes to a stop, and the operational flow returns to Al 15. This 
decision processing is repeatedly conducted until the power supply of the moving 

15 body detecting apparatus 1 1 is cut off. 

As described above, the moving body detecting apparatus 1 1 detects, as an 
intruder, only a pixel set moving from a monitoring region in the exterior of a 
vehicle to a monitoring region in the interior thereof and, hence, can detect only a 
body to be notified without detecting an object similar in feature to the body to be 

20 notified, such as a passer with respect to a thief. 

FIG. 8 is a block diagram showing another internal configuration of the 
moving body detecting apparatus 1 1 according to the first embodiment of the 
present invention. This moving body detecting apparatus 1 1 is mounted in a 
vehicle and functions as an antitheft apparatus. The same reference numerals 

25 represent the same or corresponding parts, and the detailed description will be 

omitted for brevity. As FIG. 8 shows, this moving body detecting apparatus 1 1 
is made up of a camera 13, an image storage control unit 55, a difference 
calculating unit 21, an image composing unit 45, an image processing unit 49, a 
recognizing unit 37, a control unit 39, a background image memory 17, a 
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difference image memory 23, an image composing memory 47, an image 
processing memory 50, and a speaker 41 . As mentioned above, of these 
components, each of the image storage control unit 55, the difference calculating 
unit 21, the image composing unit 45, the image processing unit 49, the 
recognizing unit 37 and the control unit 39 is constructed as an electronic circuit 
comprising a CPU, a ROM, a RAM and others. The background image memory 
17, the present image memory 19, the difference image memory 23, the image 
composing memory 47 and the image processing memory 50 are constructed 
using a semiconductor memory such as DRAM or ADRAM. Each of these 
memories can individually be constructed with a semiconductor memory, or all of 
these memories can also be constructed with one or plural semiconductor 
memories. 

The camera 13 is attached to a central portion of a ceiling in the interior 
of a vehicle, the image storage control unit 55 has a function to acquire static 
images from the camera 13 and to store them in the background memory 1 7 or the 
present image memory 19. The difference calculating unit 21 reads out images 
from the background image memory 17 and the present image memory to produce 
a difference image and put it in the difference image memory 23. The image 
composing unit 45 reads out a difference image from the difference image 
memory 23 to process it with a composite image stored in the image composing 
memory 47, and then stores the processed image in the image composing memory 
47. 

The image processing unit 49 reads out a composite image from the image 
composing memory 47 to conduct various image processing thereon and then 
stores the resultant image in the image processing memory 50. The recognizing 
unit 37 recognizes whether or not a moving body exists in the image stored in the 
image processing memory 50. 
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The image composing memory 47 is for storing a composite image, and 
the image processing memory 50 is for storing an image which has undergone 
image processing. 

The control unit 39 has a function to generally control the image storage 
5 control unit 55, the difference calculating unit 21, the image composing unit 45, 
the image processing unit 49 and the recognizing unit 37. The camera 13 
corresponds to the image acquiring means, and the image storage control unit 55, 
the difference calculating unit 21, the image composing unit 45, the image 
processing unit 49, the recognizing unit 37 and the control unit 39 correspond to 

10 the decision means, and the speaker 41 corresponds to the notifying means. 

Secondly, referring to a flow chart of FIG. 9, a description will be given 
hereinbelow of the decision processing to be implemented in the control unit 39. 
The implementation of this decision processing starts in response to the power-on 
of the moving body detecting apparatus 1 1 . 

15 First of all, in S210, an instruction is given to the image storage control 

unit 55 for initializing the present image memory 19. Upon receipt of this 
instruction, the image storage control unit 55 initializes the present image memory 
1 9. The image storage control unit 55 makes a reply on the completion of the 
initialization to the control unit 39, and in response to this reply, the control unit 

20 39 implements the following steps. Although the description will be omitted in 
the following steps, as with this step, when receiving various types of instructions, 
each of the units (the image storage control unit 55, and others) makes a reply on 
the completion of the instruction to the control unit 39 and, upon receipt of the 
replay, the control unit 39 executes the following steps. 

25 In S215, an instruction is given to the image storage control unit 55 for 

receiving static images from the camera 13 to store them in the present image 
memory 19. Upon receipt of this instruction, the image storage control unit 55 
receives static images from the camera 13 and stores them in the present image 
memory 19. 



27 



In S220, an instruction is given to the difference calculating unit 21 for 
producing a difference image. In response to this instruction, with respect to 
each pixel, the difference calculating unit 21 calculates the absolute value of a 
difference in luminance value between a background image stored in the 
background image memory 17 and a static image stored in the present image 
memory 19 to produce a difference image in a manner such that the calculated 
value is used as a pixel value, with the produced difference image being stored in 
the difference image memory 23. 

In S223, an instruction is given to the image composing unit 45 for 
initializing the image composing memory 47. Upon receipt of this instruction, 
the image composing unit 45 initializes the image composing memory 47. 

In S225, an instruction is given to the image composing unit 45 for 
transferring the difference image to the image composing memory 47. In 
response to this instruction, the image composing unit 45 reads out the difference 
image from the difference image memory 23 and stores it as a composite image in 
the image composing memory 47. This composite image is identical to the 
difference image. 

In S230, an instruction is given to the image storage control unit 55 for the 
initialization of the present image memory 1 9. This step is the same as the 
above-mentioned S2 1 0 . 

In S235, an instruction is given to the image storage control unit 55 for 
receiving a static image from the camera 13 and for storing it in the present image 
memory 19. The processing in this step is the same as that in S215. 

In S240, an instruction is given to the difference calculating unit 21 for 
producing a difference image. The processing in this step is the same as that in 
S220. 

In S245, an instruction is given so that the value of a counter to be used for 
the decision in S250, which will be mentioned later, is incremented by 1 and the 
image composing unit 45 carries out the image composing. Upon receipt of this 
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instruction, the image composing unit 45 reads out a difference image from the 
difference image memory 23 by one pixel and a composite image from the image 
composing memory 47 by one pixel to make a comparison between the read-out 
pixel values. If the pixel value of the difference image is higher, the pixel value 
5 stored in the image composing memory 47 is renewed with the pixel value of the 
difference image. On the other hand, if the pixel value of the difference image is 
lower, nothing is done. This processing is conducted with respect to all the 
pixels thereof. 

In S250, on the basis of the value of the counter, a decision is made as to 

10 whether or not the image composing instruction to the image composing unit 45 
in S245 has been given four times. If the image composing instruction has been 
made four times, the value of the counter is set at zero, and the operational flow 
advances to S255. On the other hand, if the image composing instruction does 
not reach four times, the operational flow returns to S230. 

15 In S255, an instruction is given to the image processing unit 49 for 

conducting image processing. Upon receipt of this instruction, the image 
processing unit 49 reads out a composite image from the image composing 
memory 47 and conducts various image processing such as binary conversion and 
labeling to be implemented before image recognition and stores it in the image 

20 processing memory 50. 

In S260, an instruction is given to the recognizing unit 37 for the 
recognition processing. Upon receipt of this instruction, the recognizing unit 37 
reads out an image from the image processing memory 50 to make a decision as 
to whether or not a moving body exists in that image, and returns the decision 

25 result to the control unit 39. This decision is made in consideration of a shape, 
area or the like of pixel set. 

According to a decision result in S265, that is, according to the decision 
result in the recognizing unit 37 in S260, the operational flow advances to 
different ways. If the decision result shows the existence of a moving body, the 
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operational flow advances to S270. On the other hand, if it shows no existence 
of a moving body, the operational flow returns to S210. 

In S270, a signal is sent to the speaker 41 to sound a siren for a 
predetermined period of time (for example, 1 minute). After the elapse of the 
predetermined period of time, the sending of the signal to the speaker 41 comes to 
a stop, and the operational flow returns to S210. In this way, the decision 
processing is repeatedly conducted until the power supply to the moving body 
detecting apparatus 1 1 is cut off. 

In addition, a description will be given hereinbelow of the concrete 
example of the above-mentioned decision processing. 

FIG. 10A shows images 101 to 110 forming difference images produced 
from static images taken by the camera 1 3 and a background image. These 
difference images 101 to 1 10 are arranged from the left side to the right side to 
continue in time sequence. FIG. 1 OB is a table showing pixel values of pixels A 
of the images 101 to 1 10. 

First, the images 101 to 105 successively undergo image composing 
processing to produce a composite image (see S21 1 to S250 in FIG. 9). This 
image composing is successively done in a manner such that, with respect to each 
pixel, a pixel value are compared with another pixel value to select the higher 
pixel value. Concretely, noting the pixels A in the images 101 to 105, as the 
pixel value of the pixel A of a composite image obtained from these five images, 
the pixel value "98" of the image 104 is at a maximum as shown in FIG. 10B and, 
hence, this highest value "98" is selected. 

The image composing is made on the images 101 to 105 and the resultant 
composite image undergoes the image processing (see S255 in FIG. 9) and the 
recognition processing (see S260 in FIG. 9). After the completion of these 
processing, the image composing is newly made with respect to the images 106 to 
110. 



30 



FIG. 1 1 is an illustration of a background image 1201 showing a driver's 
seat side door and its vicinity. FIG. 12A is an illustration of an n-th present 
image 1203, showing a state in which an arm B breaks in the interior of a vehicle. 
FIG. 12B is an illustration of an n+4-th present image 1205, showing a state in 
5 which the arm B enters a lower portion of the interior of the vehicle. FIG. 1 3 A 
is an illustration of a difference image 1207 produced from the background image 
1201 shown in FIG. 1 1 and the n-th present image 1203 shown in FIG. 12A, 
showing pixel sets C different in luminance value from the background image 
1201. FIG. 13B is an illustration of a difference image 1209 produced from the 
10 background image shown in FIG. 1 1 and the n+4-th present image 1205 shown in 
FIG. 12B, showing pixel sets D different in luminance value from the background 
image 1201. 

FIG. 14 is an illustration of a composite image 1211 obtained from the 
n-th difference image 1207 shown in FIG. 13 A, the n+l-th, n+2-th and n+3-th 
15 difference images (not shown) and the n+4-th difference image 1209 shown in 
FIG. 13B. As obvious from this illustration, pixel sets E have a larger area as 
compared with the pixel sets C shown in FIG. 13 A and the pixel sets D shown in 
FIG. 13B. 

Thus, the moving body detecting apparatus 1 1 makes a decision on the 
20 existence of a moving body through the use of a composite image in which a 

movement trajectory of the body appears. This enables the detection of a small 
moving body with extremely higher accuracy than that of a conventional 
technique. 

Moreover, since a difference image is produced by calculating the absolute 
25 value of the difference in luminance value between a present image taken by the 
camera 13 and a background image stored in advance and a composite image is 
produced using that difference image, even if a moving body has a lower 
luminance as compared with a background image, the moving body is detectable 
with high accuracy. 



31 



As described above, according to this embodiment, in the moving body 
detecting apparatus, the image acquiring means acquires static images continuing 
in time sequence within the same monitoring range, and the decision processing 
means receives the static images from the image acquiring means to produce a 

5 composite image by successively conducting the image composing on the static 
images so that each pixel has the highest luminance value and, when the number 
of static images used for the image composing reaches a predetermined value, 
makes a decision as to whether or not a moving body exists in the composite 
image. The image acquiring means is realizable through the use of a 

1 0 photographing (imaging) device such as a camera, or it is also realizable with an 
interface device which merely obtains images picked up by a device separate from 
the moving body detecting apparatus. The "luminance value" may signify a 
simple brightness in the case of a monochrome image and, in the case of a color 
image, may signify an independent luminance value of each of RGB colors or the 

15 result luminance value of the three colors. 

In a conventional technique, a background image including no moving 
body and a static image picked up are compared one by one to detect signs of a 
moving body and this detection processing is conducted with respect to a plurality 
of static images to track the positions of the signs of a moving body in the static 

20 images for making a decision as to whether or not it is a moving body. 

Therefore, in a case in which a moving body is relatively small in the entire static 
image, the conventional technique can result in misjudgment. On the other hand, 
according to the above-described embodiment, prior to the detection of a body, 
the decision processing means conducts the image composing on the static images 

25 so that each pixel has the highest luminance value, and detects a moving body 
using one composite image. A movement trajectory of a moving body appears 
in the composite image and the moving body appears in a state made larger than 
the original size of the moving body. Therefore, the moving body detecting 
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apparatus according to this embodiment can detect, with extremely high accuracy, 
a moving body which can fall into a background like a small moving body. 

Meanwhile, in a case in which a moving body has a lower luminance as 
compared with that of a background, when a composite image is produced by 
conflating a plurality of static images, a moving body can fall into a background. 
For example, in the case of a monochrome image, when the luminance value of 
the background is high and the luminance value of the moving body is low, even 
if the moving body exists at a given pixel, the luminance value of the background 
achieves domination over the luminance value of the moving body when the 
moving body moves and, hence, the moving body can disappear as a result. 
Therefore, in a case in which the detection of a moving body takes place in such a 
situation, the decision processing means receives static images from the image 
acquiring means and calculates an absolute value of the difference in luminance 
value between each of a plurality of static images from the image acquiring means 
and the background image stored in advance for each pixel to produce a 
difference image, and through the use of the difference images, a composite image 
is produced by the successive image composing so that each pixel has the highest 
absolute value, and when the number of difference images used for the image 
composing reaches a predetermined value, a decision is made as to whether or not 
a body moving exists in the composite image. The "background image" 
signifies an image in which a moving body is absent. 

This difference image is produced by calculating the absolute value of the 
difference in luminance value between a static image received from the image 
acquiring means and a background image stored in advance and, hence, the 
difference image does not represent the comparison result in magnitude of 
luminance between both the images but signifying the comparison result in the 
difference of luminance therebetween. Therefore, this eliminates the 
above-mentioned problem of a moving body being broken because the 
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background luminance value is high, thereby enabling high-accuracy detection of 
a moving body. 

Furthermore, as the implementation timing of the decision processing, it is 
considered that the decision processing is simply conducted repeatedly without 
5 particularly paying attention to timing. In this case, it is realizable with a simple 
arrangement, and since all the static images picked up can be put to use without 
being abandoned, the reliability of the moving body detection is improvable. 

Furthermore, although the decision processing means can carry out the 
image composing using all the static images from the image acquiring means, it is 
10 also appropriate that, when receiving static images from the image acquiring 

means, the decision processing means makes a decision as to whether to select the 
static images and continues the decision processing only in the case of the 
selection thereof. As the selecting manner, for example, it can be considered to 
use a method of regularly and selectively thinning static images or a method of 
15 making the selection only when there is a difference between the static images 
continuing in time sequence. 

Thus, the decision processing means can achieve an efficient decision 
without depending on the static image acquiring rate per unit time in the image 
acquiring means. 

20 In this connection, in the decision means, the concrete decision as to 

whether or not a moving body exists in a composite image can be as follows. 
That is, a composite image is converted into binary image data and labeled and, 
on the basis of an area and shape of the resultant pixel set, a decision is made on 
the existence of a moving body. In this case, the "on the basis of an area and 

25 shape of a pixel set" signifies that, for example, noting the areas of the pixel sets, 
pixel sets having an area exceeding a predetermined area are selected. A moving 
direction is estimated on the basis of the direction in which the selected pixel set 
is elongated and, finally, a decision is made as to whether or not it is a moving 
body. 
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Accordingly, it is possible to suppress the influence of noise existing in 
images acquired by the image acquiring means and, hence, to detect the existence 
of a moving body with high accuracy. 

It is preferable that, simultaneously with the labeling, an area of a pixel set 
5 is calculated through the use of a counter. 

Thus, the decision as to whether a moving body exists in a composite 
image is more efficiently achievable. 

Still furthermore, although the notifying means can be made to transmit 
the fact of the decision indicating the existence of a moving body to an internal 
10 separate apparatus, it is also appropriate that notifying means is provided which is 
for notifying the existence of a moving body to a user, and the decision means is 
made to give an instruction to the notifying means for notifying the existence of 
the moving body to the user. 

This enables the moving body detecting apparatus itself to inform a user of 
1 5 the existence of a moving body. 

Yet furthermore, it is preferable that the moving body detecting apparatus 
is used as an antitheft apparatus. Since the consumable power is limited during 
the vehicle storage, difficulty can be experienced in locating a sufficient light 
source(s), thus making it difficult to provide a static image having a sufficient 
20 contrast. Even in such a situation, the moving body detecting apparatus 

according to the present invention can achieve the detection with high accuracy, 
thus providing a high utility value. 

In addition, the decision processing means of the moving body detecting 
apparatus is also realizable in a manner such that a program functioning as the 
25 decision processing means is used and implemented in a computer incorporated 
into the moving body detecting apparatus. This program is recorded on a 
computer-readable recording medium such as a flexible disk, magnetic optical 
disk, CD-ROM, hard disk, ROM or RAM, and is loaded into the computer and 



35 



activated when needed. It is also possible to perform the loading and activation 
through a network. 

FIG. 15 is a block diagram showing the entire configuration of a 
vehicle-mounted body detecting apparatus for detecting an body existing on a 
5 load at the rear of a vehicle or a mark drawn on a road on the basis of a plurality 
of input images (monochrome images) picked up continuously toward the rear of 
the vehicle. 

As FIG. 15 shows, a body detecting apparatus 1 la is made up of an image 
pickup device (camera) 13 for photographing the rear of a vehicle, an image 

10 composing unit 45 for carrying out the image composing processing on a plurality 
of input images picked up continuously by the image pickup device 13, a 
composite image processing unit 104 functioning as composite image processing 
means and comprises a starting-point extracting unit 131 for extracting, as a 
starting point, a feature point (which is equally referred to as an image pickup 

15 point) of a body from the input image at the start of the image composing 

processing and a displacement data producing unit 121 for obtaining displacement 
data (in this case, the coordinates of a starting point and end point of a trajectory, 
the length of the trajectory) indicative of a displacement of a static body appearing 
in the input image on the basis of the starting point extracted by the starting-point 

20 extracting unit 131 and a composite image produced by the image composing unit 
45, a movement distance calculating unit 105 for detecting the distance of the 
movement of the vehicle while the image pickup device 13 picks up the images to 
be used in the image composing unit 45, and a relative position calculating unit 
106 for obtaining the position of the photographed static body on the basis of the 

25 displacement data obtained in the composite image processing unit 1 04, the 

movement data forming the detection result in the movement distance calculating 
unit 105 and mounting data (focal length, mounting height, mounting angle, and 
others) indicative of a characteristic or mounting state of the image pickup device 
45. 
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The image pickup device 1 3 is composed of a CCD camera or the like, and 
is set so that, as shown in FIG. 16A, its optical axis is directed obliquely and 
downwardly and, as shown in FIG. 16B, an infinity point (FOE) in an image 
picked up is placed at a central portion of a frame width in a horizontal direction 
5 and at an upper portion outside the frame in a vertical direction. 

The image composing unit 45 is composed of a composite image memory 
113 serving as a composite image storing means to store a composite image by 
one frame and a comparator 111 serving as a pixel value updating means to, with 
respect to each pixel existing at the same position in a frame, make a comparison 

10 in magnitude of the pixel value between a composite image stored in the 

composite image memory 113 and a monochrome image (which will be referred 
to hereinafter as an "input image") from the image pickup device 13 for updating 
the pixel value of the composite image when the pixel value of the input image is 
higher. However, the comparator 1 1 1 is made to directly store the input image at 

15 the start of image composing in the composite image memory 113. 

In addition, the starting-point extracting unit 131 constituting the 
composite image processing unit 104 is composed of an edge extracting filter 133 
for extracting an edge of an input image at the start of the image composing and a 
starting-point information database 135 for storing, as starting-point information, 

20 the coordinates of starting points in a frame in a state where pixels corresponding 
to the edge (image pickup point group) extracted by the edge extracting filter 133 
are taken as the starting points. 

The displacement data producing unit 121 is composed of a notice line 
setting unit 127 for setting, as a notice line, each of lines making connections 

25 between the starting points stored in the starting-point information database 135 
and the FOE in an input image, an edge extracting filter 123 for, on the basis of a 
composite image stored in the composite image memory 113 after the image 
composing, extracting pixels corresponding to an edge of the composite image 
and existing on the notice lines as end points, and a trajectory length calculating 
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unit 125 for combining the coordinates of the starting point and the end point 
existing on the same notice line to obtain, as a trajectory length, the distance 
between the starting point and the end point after the combination for producing 
displacement data including the coordinates and the trajectory length. 
5 FIG. 17A is an illustration of one example of an image to be taken by the 

image pickup device 13 in a case in which a vehicle is parked in a parking zone in 
a backward-looking posture, where there appear partition lines of a parking space 
and a tire stopper made using a concrete block. 

When this image is supplied as an input image at the start of image 

10 composing to the starting-point extracting unit 13 1, as shown in FIG. 17B the 

edge extracting filter 133 extracts profiles of the partition lines and a profile of the 
tire stopper which have a higher contrast with respect to other portions, and the 
coordinates of the pixels corresponding to these profiles are stored as 
starting-point information in the starting-point information database 135. 

15 Moreover, the image composing unit 45 sequentially conflates a plurality 

of input images (see FIG. 18 A) picked up by the image pickup device 13 in 
accordance with the backward movement of the vehicle to provide a composite 
image as shown in FIG. 18B. A trajectory of an image pickup point having a 
high gradient appears in that composite image, and the trajectory is along a notice 

20 line directed at the FOE. 

FIG. 18C is a distribution map of pixel values of pixels on the notice line 
L shown in FIG. 18B. In this case, let it be assumed that two starting points A 
and B exist on the notice line L. 

The edge extracting filter 123 extracts an edge (that is, a spot at which the 

25 pixel value decreases rapidly) along the notice line L from each of the starting 

points A and B, thereby extracting spots C and D as end points of the trajectory on 
the notice line L. 

In addition, the trajectory length calculating unit 125 obtains the trajectory 
length TA between the starting point A and the end point C and the trajectory 
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length TB between the starting point B and the end point D, and the displacement 
data producing unit 121 produces displacement data comprising the coordinates of 
each of the starting point A and the end point C and the trajectory length TA and 
displacement data comprising the coordinates of each of the starting point B and 

5 the end point D and the trajectory length TB. 

The displacement data producing unit 121 conducts the same processing 
with respect to all the notice lines including starting points stored in the 
starting-point information database 135, with all the displacement data thus 
obtained being supplied to the position calculating unit 106. 

10 Incidentally, although, in this case, an edge of a composite image is 

extracted along a notice line to extract an end point of a trajectory, it is also 
appropriate that a region including a starting point stored in the starting-point 
information database 135 and having a pixel value larger than a threshold set in 
advance is extracted from a distribution map (see FIG. 18C) of pixel values along 

15 a notice line and the length of the extracted region is obtained as a trajectory 
length. 

The position calculating unit 106 calculates a position (relative distance, 
bearing, height from a road surface, and others) of a static body appearing in an 
input image on the basis of the displacement data (coordinates of starting points 

20 and end points of loci, trajectory lengths) thus obtained, the movement data on a 
vehicle (that is, the image pickup device 13) and the mounting data (focal length, 
mounting height, mounting angle) on the image pickup device 13 set in advance 
through the use of a well-known method (for example, see Japanese Patent 
Laid-OpenNos. HEI 7-78252 and 2001-187553) based on a tigonometrical 

25 survey. 

As described above, the body detecting apparatus 1 la conducts the image 
composing on input images taken continuously and obtains displacement data 
indicative of a displacement of a static body in a frame in the input images on the 
basis of a trajectory of an image pickup point appearing in the resultant composite 
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image and a starting point extracted from the input image at the start of the image 
composing. 

That is, with this body detecting apparatus 1 la, it is possible to associate 
the same image pickup points shown in different images in a less calculation 
5 quantity without employing the pattern matching requiring a fabulous processing 
quantity and to obtain displacement data indicative of the displacement with high 
accuracy. As a result, this enables the detection of the position of a static body 
appearing in input images for a short time, and achieves high-reliability detection. 
As described above, according to this embodiment, there is provided a 

1 0 displacement data extracting method of obtaining a plurality of input images 
continuously picked up and acquiring, as each of representative pixel values, a 
maximum value of pixels existing at the same positions in frames of the input 
images to produce a composite image comprising the representative pixel values 
and extracting displacement data indicative of a displacement of a photographed 

1 5 body in the frames on the basis of a trajectory of an image pickup point appearing 
in the composite image. 

That is, in the case of a conventional technique (for example, Japanese 
Patent Laid-Open No. HEI 7-78252), although the distance between image pickup 
points (for example, edges of a body or the like) to be associated with each other 

20 becomes short as shown in FIG. 27A, eventually for specifying the association 
(correspondence), there is a need to carry out the pattern matching requiring a 
fabulous calculation quantity between input images. On the other hand, 
according to this embodiment, a trajectory of an image pickup point to be 
associated appears in a composite image and the association in image pickup point 

25 between the input images can clearly be seized on the basis of the trajectory, 
which can eliminate the need for the pattern matching. 

For example, as the displacement data, it is also possible to use the 
coordinates of a starting point and end point of a trajectory of an image pickup 
point or to use a length of the trajectory. 
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Furthermore, as described above, there is provided a body detecting 
apparatus comprising image composing means for obtaining a plurality of input 
images continuously picked up and acquiring, as each of representative pixel 
values, a maximum value of pixels existing at the same positions in frames of the 

5 input images to produce a composite image comprising the representative pixel 
values and composite image processing means for extracting displacement data 
indicative of a displacement of a photographed body in the frames on the basis of 
a trajectory of an image pickup point appearing in the composite image. 

That is, this body detecting apparatus realizes the above-mentioned 

10 displacement data extracting method, and provides the same effects. 

The image pickup device can be fixed in place or can also be mounted on 
a mobile body (moving body). In the case of the image pickup device being 
fixed in place, a movement quantity of a mobile body can be obtained on the basis 
of the displacement data extracted by the composite image processing means. 

15 On the other hand, in the case of the image pickup device being mounted on a 
mobile body, a position of a static body can be obtained on the basis of distance 
data indicative of a movement distance of the mobile body in the middle of the 
photograph of the input images used for the production of the composite image, 
the displacement data extracted by the composite image processing means and the 

20 mounting position data indicative of the mounting position of the image pickup 
device. 

Meanwhile, in a case in which mobile body is going straight along a line 
directed at the motion infinity point (FOE : Focus Of Expansion) in the input 
image, an image pickup point on a static body photographed as an input image 
25 forms a trajectory along a notice line in a composite image. 

Therefore, the composite image processing means extracts, as a starting 
point (see "S" in FIG. 28A), an edge of a body taken in an input image at the start 
of image composing in the image composing means and extracts, as an end point 
(see "E" in FIG. 28B), an edge of a trajectory shown in a composite image 
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produced by the image composing means, and a line passing through a motion 
infinity point in an input image is set as a notice line (see "L" in FIG. 28 A and 
28B) and the displacement data is obtained on the basis of a starting point and an 
end point existing on the same notice line. 
5 Moreover, in this case, if the image composing means performs the image 

composing on only the pixels on the notice line, needed for obtaining the 
displacement data, the processing quantity in the image composing means is 
considerably reducible. 

However, since there is a possibility that a plurality of starting points 

10 exists on the same notice line, it is preferable that the image composing means 
controls the timing of the completion of image composing according to starting 
point to prevent the loci different in starting point from each other from being 
connected to each other. 

Furthermore, the image composing means includes composite image 

15 storing means, and pixel updating means sequentially compares a pixel value of 
an input image obtained from the image pickup device and a pixel value of a 
composite image stored in the composite image storing means and, if the pixel 
value of the input image is larger than the pixel value of the composite image, 
updates the storage contents in the composite image storing means with the pixel 

20 value of the input image. 

In this case, since the pixel value of the input image obtained from the 
image pickup device is temporarily preserved, it is possible to reduce the storage 
capacity and to speed up the composing processing. 

Furthermore, the mounting height of the image pickup device from a road 

25 surface varies due to a variation of the posture of the vehicle or sinking of the 

vehicle body. Therefore, in a case in which there is a need to calculate the height 
of an image pickup point from a read surface on the basis of displacement data, it 
is preferable to make a correction for offsetting the variation of the mounting 
height. 
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Still furthermore, in a case in which the image pickup device is placed so 
that the motion infinity point in an input image exists within a frame, a body at the 
infinity becomes detectable, thus enlarging the detection range. However, if the 
motion infinity point is positioned in the vicinity of a central portion of the frame, 
5 the notice line extending to the motion infinity point becomes shorter and, hence, 
difficulty is encountered in sufficiently exhibiting the distance resolution an input 
image has. 

For this reason, it is preferable that the image pickup device is placed so 
that the motion infinity point is positioned at a circumferential edge of a frame. 

1 0 However, in a case in which there is no need to detect a body at the 

infinity and it is possible to limit the body detection range, the image pickup 
device can be placed so that the motion infinity point is positioned in the exterior 
of the frame. Moreover, when the image pickup device is placed so that the 
optical axis coincides with a vertical direction (that is, the motion infinity point is 

15 positioned at the infinity), it is possible to exhibit the maximum distance 
resolution of an input image. 
Second Embodiment 

Although the above-described moving body detecting apparatus according 
to the first embodiment acts as an antitheft apparatus to be mounted in a vehicle, it 

20 is also applicable to a break-in-prevention apparatus to be installed at a place, 
such as an entrance or window of a building, where it is required to prevent an 
intruder from breaking in. Moreover, it is also applicable to an antitheft 
apparatus to be installed at a spot such as a store or art museum requiring the 
prevention of approach of a thief around an article or picture. As one example, 

25 the camera 13 is set to provide static images of an interior of an art museum and 

the interior of an exhibition room is set as an allowable area while an exhibit, such 
as a picture in the room, and its vicinities is set as an unallowable area. 

In the above-described moving body detecting apparatus 1 1 according to 
the first embodiment, in a case in which a moving body around a vehicle exists 
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continuously for a predetermined period of time, warning can be made to that 
moving body. Concretely, in a case in which the motion detecting unit 33 
detects that a pixel set residing in common to the previous frame and the present 
frame is in an allowable area, if the pixel set exists continuously in the allowable 
5 area for a predetermined period of time (for example, several seconds), a warning 
is given to that moving body, such as 1) a warning being made with a voice from 
the speaker 41, 2) an alarm lamp being lighted, and 3) a buzzer being sounded. 
This can previously prevent a moving body around the vehicle from breaking in 
the vehicle. 

10 In addition, in the above-described moving body detecting apparatus 1 1 

according to the first embodiment, in a case in which the recognizing unit 37 
judges that an intruder from the exterior of a vehicle to the interior thereof exists, 
information to the effect of the existence of the intruder can be sent through a 
transmitting means to an external different apparatus such as a portable telephone 

1 5 pertaining to the owner of the vehicle or a telephone in a house of the owner of 

the vehicle. This achieves quick notification on the existence of an intruder from 
the exterior of the vehicle to the interior thereof to the owner of the vehicle. 

Still additionally, in the above-described moving body detecting apparatus 
1 1 according to the first embodiment, in a case in which the recognizing unit 37 

20 judges that an intruder from the exterior of a vehicle to the interior thereof exists, 
static images in which the intruder appears is preserved in the present image 
memory 19. This can record the fact that valuables such as a purse were stolen 
from the interior of the vehicle. 

Moreover, although in the above-described embodiment the background 

25 image memory 17 is initialized at the start of the decision processing and 

background images are stored in the initialized background image memory 17, it 
is also appropriate that, for example, the background images are updated when the 
decision in the recognizing unit 37 indicating no existence of an intruder is made a 
predetermined number of times. 
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Still moreover, although in the above-described embodiment the 
monitoring regions "201" and "203" for monitoring window portions in static 
images are set as an "allowable area" while the monitoring regions "202" and 
"204" for monitoring door body portions in the static images are set as an 
"unallowable area" as shown in FIG. 2B, the present invention is not limited to 
this but it is also appropriate that, for example, the monitoring regions "201" and 
"203" are set as the unallowable area while the monitoring regions "202" and 
"204" are set as the allowable area, thus detecting a body such as a handbag or 
purse brought from the interior of the vehicle to the exterior thereof. In this case, 
the unallowable areas are positioned above the allowable areas. Concretely, in 
the decision processing to be implemented in the recognizing unit 37, of the 
above-mentioned conditions (1) to (3), the condition (2) is altered to the following 
condition (2'). That is, when the following condition is satisfied, a decision is 
made that a body such as a handbag or purse is moved from a door body portion 
of a vehicle to a window portion thereof. 

the area of a monitoring region > a fourth threshold 

... <2M) 

the direction of a motion vector is upward — (2'-2) 
The fourth threshold corresponds to an area of a pixel set determined to 
detect a body such as a handbag or purse reflected in the door body portion of the 
vehicle and is prescribed in advance through experiments or the like. If there is a 
pixel set satisfying all the above-mentioned conditions (1), (2') and (3), a decision 
is made that this pixel set represents a body such as a handbag or purse. This 
enables the detection of a body such as a handbag or purse taken out from the 
interior of a vehicle to the exterior thereof. 

Furthermore, although in the above-described embodiment a static image 
is divided into a vehicle interior image and a vehicle exterior image, and is stored 
in the background image memory 17 or the present image memory 19, the present 
invention is not limited to this, but it is also appropriate that a monitoring region 
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extracted from a static image is divided into a vehicle interior image and a vehicle 
exterior image, and is stored in the background image memory 1 7 or the present 
image memory 19. In this case, since only the monitoring area is stored in each 
memory, the memory size of the storage unit is reducible as compared with the 
5 case of storing a static image. 

Still furthermore, although in the above-described embodiment the pixel 
signals in the monitoring regions "20 1" to "204" are encoded with the numerical 
value "1" and the pixel signals in the other regions are encoded with the numerical 
value "0", the present invention is not limited to this, but it is also appropriate that 

10 the encoding is made with a different numerical value for each monitoring region. 
Concretely, the encoding (multi-valued conversion) is made with the numerical 
value "1" for the monitoring region "201", the encoding is made with the 
numerical value "2" for the monitoring region "202", the encoding is made with 
the numerical value "3" for the monitoring region "203", the encoding is made 

15 with the numerical value "4" for the monitoring region "204", and the encoding is 
made with the numerical value "0" for the non-monitoring region. 

In addition, although in the above-described embodiment each of the 
monitoring regions is set in the form of a point set such as bit map data, the 
present invention is not limited to this, but it is also appropriate that the 

20 monitoring regions are prescribed using the boundary (border) lines of their outer 
circumferences. These boundary lines can be expressed in the form of a point 
set or using a function. In this case, the data quantity for setting the monitoring 
regions becomes only data for setting the boundary lines and, hence, the data 
quantity is reducible as compared with the above-mentioned case of setting the 

25 monitoring regions in the form of a point set, thus reducing the memory size of 
the storage unit for storing the set data. 

Still additionally, for example, in a case in which there is no need to 
distinguish strictly between the vehicle interior and the vehicle exterior, the 
monitoring regions can also be set as circumscribed rectangles. In this case, if 
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the monitoring regions are set at both end portions of a diagonal line of the 
rectangle, the data quantity is further reducible. As one example, as shown in 
FIG. 2C, the monitoring region "203" is set in the form of a rectangle in which the 
coordinate 1 (200, 20) and the coordinate 2 (400, 60) constitute both ends of a 
5 diagonal line. Each side of the rectangle forms a boundary line of a monitoring 
region. 

Third Embodiment 

Referring to FIGs. 8,19 and 20, a description will be given hereinbelow of 
a moving body detecting apparatus 401 according to a third embodiment of the 

10 present invention. The description will basically be made about the differences 
from the above-described moving body detecting apparatus 1 1 for brevity. 

A control unit 437 is made to implement preliminary decision processing 
and decision processing, and the preliminary decision processing starts in 
response to the power-on of the moving body detecting apparatus 401, and the 

15 decision processing starts in the middle of the preliminary decision processing. 

The preliminary decision processing will first be described with reference 
to a flow chart of FIG. 19. 

In S3 10, an instruction is given to the image storage control unit 55 for 
initializing the present image memory 19. Upon receipt of this instruction, the 

20 image storage control unit 55 initializes the present image memory 19. After the 
completion of the initialization, the image storage control unit 55 makes a reply 
on the fact of the initialization completion to the control unit 437, and upon 
receipt of this reply, the control unit 437 executes the following steps. Although 
the description will be omitted in the following steps, as with this step, when 

25 receiving various types of instructions, each of the units (the image storage 

control unit 55, and others) makes a replay on the completion of that instruction to 
the control unit 437, and upon receipt of the replay, the control unit 437 executes 
the following steps. 
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In S3 15, an instruction is given to the image storage control unit 55 for 
receiving static images from the camera 13 to store them in the present image 
memory 19. Upon receipt of this instruction, the image storage control unit 55 
acquires static images from the camera 13 and puts them in the present image 

5 memory 19. 

In S320, an instruction is given to the difference calculating unit 21. 
Upon receipt of this instruction, the difference calculating unit 21 calculates, for 
each pixel, the absolute value of the difference in luminance value between a 
background image stored in the background memory 17 and a static image stored 

10 in the present image memory 19 to produce a difference image in a manner such 
that the calculated value is set as a pixel value, with the produced difference 
image being stored in the difference image memory 23. 

In S325, an instruction is given to the difference calculating unit 21 for 
making a decision as to whether a predetermined quantity of pixel set exists in the 

15 difference image. Upon receipt of this instruction, the difference calculating unit 
21 makes a decision as to whether, in the calculated difference image, a pixel set 
having a predetermined quantity exists at a window portion. In this case, the 
"predetermined quantity" signifies a magnitude or size of a pixel set appearing 
when a person stands in the vicinity of a door. If there is a pixel set having the 

20 predetermined quantity, the operational flow advances to S3 30. Otherwise, the 
operational flow proceeds to S340. 

The operational flow from S3 30 depends on whether or not the decision 
processing, which will be mentioned later, is in execution. If the decision 
processing is in execution, the operational flow returns to S3 10. Otherwise, the 

25 operational flow proceeds to S335. In S335, the decision processing is started and, 
after the starting, the operational flow returns to S3 10. 

The operational flow from S340 depends on whether or not the decision 
processing is in execution. If the decision processing is in execution, the 
operational flow proceeds to S345. Otherwise, the operational flow returns to 
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S3 10. In S345, the decision processing is forcibly stopped, and then the 
operational flow returns to S3 10. 

Secondly, the decision processing will be described hereinbelow with 
reference to a flow chart of FIG. 20. 

In S410, an instruction is given to the image composing unit 45 for the 
initialization of the image composing memory 47. Upon receipt of this 
instruction, the image composing unit 45 initializes the image composing memory 
47. 

In S415, an instruction is given to the image composing unit 45 for 
transferring a difference image to the image composing memory 47. In response 
to this instruction, image composing unit 45 reads out a difference image from the 
difference image memory 23 and stores it as a composite image in the image 
composing memory 47. This composite image is identical to the difference 
image. 

In S420, an instruction is given to the difference calculating unit 21 for 
checking whether or not a newer difference image than the difference image 
transferred in S415 exists in the difference image memory 23. In response to 
this instruction, the difference calculating unit 21 makes a decision as to whether 
or not a newer difference image exists in the difference image memory 23, and 
makes a reply on that decision result to the control unit 437. If the newer 
difference image exists in the difference image memory 23, the operational flow 
advances to S425. Otherwise, the operation falls into the stand-by condition 
until a new difference image is stored in the difference image memory 23. 

In S425, the value of a counter to be used for the decision in S430, which 
will be mentioned later, is incremented by one, and an instruction is given to the 
image composing unit 45 for the image composing. In response to this 
instruction, the image composing unit 45 reads out, pixel by pixel, a difference 
image from the difference image memory 23 and a composite image from the 
image composing memory 47 to make a comparison between the pixel values. If 
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the pixel value of the difference image is higher, the pixel value of the pixel stored 
in the image composing memory 47 and compared is updated with the pixel value 
of the difference image. On the other hand, if the pixel value of the difference 
image is lower, nothing is done. This processing is conducted with respect to all 
5 the pixels. 

In S430, on the basis of the counter value, a decision is made as to whether 
or not the image composing instruction to the image composing unit 45 in S425 
has been given four times. If the image composing instruction has been made 
four times, the value of the counter is set at zero, and the operational flow 

10 advances to S435. On the other hand, if the image composing instruction does 
not reach four times, the operational flow returns to S420. 

In S435, an instruction is given to the image processing unit 49 for the 
image processing. Upon receipt of this instruction, the image processing unit 49 
reads out a composite image from the image composing memory 47 and conducts 

15 various image processing such as binary conversion and labeling to be 

implemented before image recognition and stores it in the image processing 
memory 50. 

In S440, an instruction is given to the recognizing unit 37 for the 
recognition processing. Upon receipt of this instruction, the recognizing unit 37 

20 reads out an image from the image processing memory 50 to make a decision as 
to whether or not a moving body exists in that image, and returns the decision 
result to the control unit 437. This decision is made in consideration of a shape, 
area or the like of pixel set. 

The operational flow from S445 depends on the decision result in the 

25 recognizing unit 37 in S440. If the decision result in S440 shows the existence 
of a moving body, the operational flow advances to S450. On the other hand, if 
the decision result in the S440 shows no existence of a moving body, the 
operational flow advances to S455. 
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In S450, a signal is sent to the speaker 41 to sound a siren for a 
predetermined period of time (for example, 1 minute). After the elapse of the 
predetermined period of time, the sending of the signal to the speaker 41 comes to 
a stop, and the operational flow returns to S455. 
5 In S455, an instruction is given to the difference calculating unit 21 for 

checking whether or not a newer difference image than the difference image in 
S420 exists in the difference image memory 23. In response to this instruction, 
the difference calculating unit 21 makes a decision as to whether or not a newer 
difference image exists in the difference image memory 23 and returns the 

10 decision result to the control unit 437. If the newer difference image exists in the 
difference image memory 23, the operational flow returns to S410. Otherwise, 
the operation falls into the stand-by condition until a new difference image is 
stored in the difference image memory 23. This decision processing is 
repeatedly conducted until the power supply of the moving body detecting 

15 apparatus 1 1 is cut off or until it is stopped by the preliminary decision 
processing. 

In the moving body detecting apparatus 401, the control unit 437, the 
image storage control unit 55 and the difference calculating unit 21 correspond to 
preliminary decision processing means, while the control unit 437, the difference 

20 calculating unit 21, the image composing unit 45, the image processing unit 49 
and the recognizing unit 37 correspond to the decision means. 

With this moving body detecting apparatus 401, since the decision 
processing is conducted only when needed, if the power per unit time to be used 
for the decision processing is larger than the power per unit time to be used for the 

25 preliminary decision processing, the power consumption per unit time is reducible 
as compared with the case in which the decision processing is carried out at all 
times. 

As described above, for the implementation timing of the decision 
processing, the preliminary decision processing is provided to compare a 
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background image stored in advance and a static image obtained by the image 
acquiring means with each other with respect to a predetermined region to make a 
preliminary decision as to whether or not they shows a difference exceeding a 
predetermined quantity and the decision means conducts the decision processing 
5 only while the preliminary decision result in the preliminary decision processing 
means indicates a difference exceeding the predetermined quantity. In this case, 
the "predetermined region" signifies a specified area in an image determined in 
advance, and the "difference exceeding the predetermined quantity" signifies a 
quantitatively calculable physical quantity, for example, the number of pixels 

10 having different luminance values (disagreement in luminance) or the discrete 
degrees of these pixels. 

In a case in which the moving body detecting apparatus is used as a 
break-in detecting apparatus for a vehicle, the image acquiring means is set so that 
a door portion and a window portion of a door are photographed to obtain static 

15 images and the window portion in the static image is set as a predetermined region 
for which the preliminary decision means makes a preliminary decision. In this 
case, the preliminary decision processing means compares the window portion in 
a background image stored in advance and the window portion in a static image 
acquired by the image acquiring means to continuously make a decision as to 

20 whether or not a difference therebetween, which indicates the possibility of 

existence of a person, exists in both the images. Moreover, the decision means 
implements a series of decision processing only while the decision in the 
preliminary decision processing means shows the "difference indicating the 
possibility of existence of a person". 

25 Thus, only when a person appears in the window portion, that is, only 

when a person exists in the vicinity of a door in the exterior of the vehicle, the 
decision processing means implements a series of decision processing. 
Accordingly, the decision processing is conducted only when needed, if the power 
per unit time to be used for the decision processing in the decision processing 
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means is larger than the power per unit time to be used for the preliminary 
decision processing in the preliminary decision processing means, the power 
consumption per unit time is reducible as compared with the case in which the 
decision processing is carried out at all times. 
Fourth Embodiment 

In the moving body detecting apparatus 401 according to the third 
embodiment, it is also appropriate that a difference image for only a required 
region, for example, a difference image for only a window portion, is produced in 
the preliminary decision processing. This can further enable the reduction of 
power consumption. However, there is a need to produce a difference image in 
the decision processing on another occasion. 

Although in the above-described embodiment a composite image is 
produced from five static images, no limitation is imposed on the number of static 
images. It can be determined on the basis of information such as a possible 
frame rate of the camera 13 or an estimated movement speed of a moving body 
forming an object of detection. 

Although in the above-described embodiment all the static images taken 
by the camera 13 are made to be put to use, it is also appropriate that a selecting 
means is provided to selectively use the static images taken by the camera 13. 
As the selecting manner, it can be considered to use a method of regularly and 
selectively thinning static images or a method of dynamically changing the 
number of static images to be used on the basis of the processing load. This 
enables the realization of the present invention even through the use of a 
processing unit such as a CPU having a low processing ability per unit time. 

It is also appropriate that, when the image processing unit 49 conducts the 
labeling processing, an area of a pixel set is simultaneously calculated through the 
use of a counter and the calculation result is handed over to the recognizing unit 
37. This can reduce the processing load on the recognizing unit 37 and can 
improve the processing efficiency as a whole. 
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Although in the above-described embodiments each of parts such as the 
control unit 39, 437, the image storage control unit 55, the difference calculating 
unit 21 and others is an electronic circuit constructed separately using a CPU, a 
ROM, a RAM and other components, it is also appropriate to realize all the parts 
5 through the use of one CPU and a program to be executed by the CPU. 

The steps in the above-described embodiments can also be implemented in 
parallel or in a difference sequence provided that it can provide the effects of the 
present invention. 
Fifth Embodiment 

10 Furthermore, a description will be given hereinbelow of a body detecting 

apparatus according to a fifth embodiment of the present invention. The 
difference of this body detecting apparatus according to the fifth embodiment 
from the body detecting apparatus according to the first embodiment is only a 
portion of the configuration of the image composing unit, and the description will 

15 principally be given about the portion different therefrom. 

That is, in this embodiment, as shown in FIG. 21, an image composing 
unit 45a is provided with an FOE displacement compensating unit 115 
functioning as a compensating means to compensate for a displacement 
(positional deviation) of the FOE (Focus Of Expansion) occurring among the 

20 input image from the image pickup device 13 on the basis of detecting signals 
from a steering sensor 141 for detecting a variation of the advancing (traveling) 
direction of a vehicle, an acceleration sensor 143 for detecting a speed variation of 
the vehicle in the advancing direction and a yaw rate sensor 145 for detecting a 
variation of the angular velocity of the vehicle in its turning direction. The input 

25 images undergoing the displacement compensation in the FOE displacement 
compensating unit 1 15 are supplied to the comparator 111. 

In a case in which the advancing direction of a vehicle varies during the 
image composing on input images as shown in FIG. 22 A, the FOE of the input 
image (which will hereinafter be referred to as a "present FOE") is horizontally 
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displaced or shifted along a horizontal plane with respect to the FOE of the input 
image at the start of the image composing (which will hereinafter be referred to as 
an "initial FOE") as shown in FIG. 22B. In the body detecting apparatus 
constructed as mentioned above, the FOE displacement compensating unit 115 
5 detects this displacement on the basis of a detection signal from the steering 
sensor 141 and, as shown in FIG. 22C, horizontally shifts the association 
(correspondence) in pixel between the composite image and the input image to 
offset the FOE displacement so that the present FOE coincides with the initial 
FOE. 

10 Moreover, in a case in which the posture of a vehicle is inclined forwardly 

or backwardly due to rapid acceleration/deceleration, the present FOE is displaced 
vertically with respect to the initial FOE. The FOE displacement compensating 
unit 115 detects this displacement on the basis of a detection signal from the 
acceleration sensor 143 and vertically shifts the association in pixel between the 

15 composite image and the input signal to compensate for the FOE displacement so 
that the present FOE agrees with the initial FOE. 

Still moreover, in a case in which a vehicle is inclined in a road surface 
width direction as shown in FIG. 23 A or it is inclined in a roll direction due to its 
turning, the input image is shifted in a turning direction with respect to the input 

20 image at the start of the image composing. The FOE displacement compensating 
unit 115 detects this displacement on the basis of a detection signal from the yaw 
rate sensor 145 and, as shown in FIG. 23B, shifts the association in pixel between 
the composite image and the input image in vertical directions, horizontal 
directions and turning directions so that the present FOE agrees with the initial 

25 FOE and the horizontal and vertical directions in the frames agree with each other. 
Owing to this compensation, the trajectory of an image pickup point 
becomes a straight line(s) along a notice line(s) as shown in FIG. 22C or 23B 
without becoming a curve as shown in FIG. 22B and, hence, a composite image 
can be processed to extract displacement data as in a case in which the vehicle is 
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going straight. That is, the position of a static body appearing in an input image 
can easily be obtained stably and accurately irrespective of the variation in 
behavior and posture of the vehicle. 

As described above, if the moving direction of the mobile body or the 

5 posture of a mobile body (in other words, the posture of the image pickup device 
mounted on the mobile body) varies in accordance with a behavior of the mobile 
body, a displacement of the motion infinity point occurs between input images to 
be used for the production of a composite image, and when this displacement 
occurs, the trajectory in the composite image deviates from the notice line and, 

10 hence, the acquisition of the displacement data such as a trajectory length 

becomes more troublesome. Therefore, the image composing means includes 
compensating means to, if a displacement of the infinity point occurs, compensate 
for the displacement by varying the association between the pixel of the input 
image obtained from the image pickup device and the pixel of the composite 

15 image stored in the composite image storing means. 

In this case, the compensating means can detect a variation of the 
advancing direction of a mobile body as the behavior of the mobile body, and in 
particular, if the mobile body is a vehicle, the compensating means can detect a 
variation of the posture of the vehicle. 

20 In addition, as the variation of the posture of the vehicle, it is possible to 

detect at least one of the inclination of the vehicle in a longitudinal direction due 
to an acceleration in the advancing direction of the vehicle and the inclination in 
transverse direction due to an angular velocity in a turning direction of the 
vehicle. 

25 Sixth Embodiment 

Although in the above-described embodiment the image pickup device 13 
is set such that the FOE is placed at a central portion of a frame width in a 
horizontal direction and at an upper portion outside the frame in a vertical 
direction, it is also appropriate that the image pickup device 13 is set such that an 
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optical axis P agrees with the direction of the FOE as shown in FIG. 24A and the 
FOE is positioned at a central portion of a frame as shown in FIG. 24B. 

However, in this case, limitation is imposed on the resolution in the 
direction indicating the height of an image pickup object or in the direction 
indicating the distance up to the image pickup object and the calculation accuracy 
lowers. For this reason, it is preferable that, for making the most of the 
resolution, the image pickup device 13 is placed such that, at least, an upper end 
of the image pickup range of the image pickup device 1 3 coincides with the FOE 
direction as shown in FIG. 25A and the FOE is positioned at an upper edge 
portion of the frame as shown in FIGs. 25B and 25C. 

Moreover, although even a body far apart therefrom can be accommodated 
in a frame in the case of the FOE being positioned in the frame in this way, if it is 
possible to limit the detection range, the image pickup device 13 can also be 
installed so that the optical axis agrees with a vertical direction as shown in FIG. 
26A and the notice lines become in parallel with each other (the FOE is positioned 
at the infinity above the frame) as shown in FIG. 26B. 

Still moreover, although in the above-described embodiments the image 
composing unit 45, 45a is made to produce a composite image using all pixels, it 
is also appropriate that a composite image is produced using only pixels on notice 
lines set by the notice line setting unit 127. In this case, the processing quantity 
in the image composing unit 45, 45a is considerably reducible. 

However, since there is a possibility that a plurality of starting points 
exists on the same notice line, it is preferable that the comparator 1 1 1 controls the 
timing of the completion of image composing according to starting point to 
prevent the loci different in starting point from each other from being connected 
to each other. 

Yet moreover, although in the above-described embodiment the position 
calculating unit 106 uses the mounting data set in advance, it is also appropriate 
that, in a case in which a vehicle body falls into a sinking condition due to the 
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loading of a heavy thing, the vehicle body sinking is detected to correct the 
mounting height forming one of mounting data. 

In addition, although in the above-described embodiment the image pickup 
device 13 is mounted on a vehicle and the detection of the position of a static 
5 body is made on the basis of a composite image made from input images 

continuously picked up by the image pickup device 13, it is also appropriate that 
the image pickup device 13 is fixed in place and the position or movement 
quantity of a moving body is obtained on the basis of displacement data obtained 
from a composite image of input images continuously taken by the image pickup 
10 device 13. 

It should be understood that the present invention is not limited to the 
above-described embodiments, and that it is intended to cover all changes and 
modifications of the embodiments of the invention herein which do not constitute 
departures from the spirit and scope of the invention. 



